Towards a General Theory of Optimal Testing

In Pericchi and Pereira (2012)it is argued against the traditional way on which testing is based on fixed significance level, either using p-values (with fixed levels of evidence, like the 5% rule) or α values. We instead, follow an approach put forward by DeGroot (1975), on which an optimal test is chosen by minimizing type I and type II errors. Morris DeGroot in his authoritative book (1975), Probability and Statistics 2nd Edition, stated that it is more reasonable to minimize a weighted sum of Type I and Type II error than to specify a value of type I error and then minimize Type II error. He showed it beyond reasonable doubt, but only in the very restrictive scenario of simple VS simple hypothesis, and it is not clear how to generalize it. We propose here a very natural generalization for composite hypothesis, by using general weight functions in the parameter space. This was also the position taken by Pereira (1985, 1993, 2010). We show, in a parallel manner to DeGroot’s proof and Pereira’s discussion, that the optimal test statistics are Bayes Factors, when the weighting functions are priors with mass on the whole parameter space. On the other hand when the weight functions are point masses in specific parameter values of practical significance, then a procedure is designed for which the sum of Type I error and Type II error in the specified points of practical significance is minimized. This can be seen as bridge between Bayesian Statistics and a new version of Hypothesis testing, more in line with statistical consistency and scientific insight. 1 Minimizing the sum of weighted errors Morris DeGroot (1975), argued that instead of fixing Type I error and minimize Type II, a better hypothesis testing paradigm is to choose the test as to minimize a weighted sum of the errors, namely Minδ [a · αθ0(δ) + b · βθ1(δ)], (1) where δ denotes the test: δ(x) = 1R(x), where R is the Rejection Region of H0, IS(y) is the indicator function, equal to 1 if and only if y ∈ S. Notice the apparently slight difference of